Repository navigation
Conversation
Switch the default Parquet data page version from V1 to V2 across C++ and Python (PyArrow). V2 data pages enable per-page compression selectivity (via min_compression_savings) and include explicit page-level null count metadata (num_nulls) in the page header. Closes apache#55272.
|
|
… masking and synchronize V2 page defaults
Reorder MaybeReplaceValidity before update_stats in WriteArrowDictionary so writeable_indices reflects parent nulls prior to computing page stats, and guard dictionary reuse with chunk_indices->null_count() == 0.
Update TestSerialize for BooleanType in Parquet C++ to account for V2 selective compression keeping incompressible 7-byte boolean pages uncompressed.
Update test_parquet_metadata_api expected boolean encodings to {'RLE'} for V2 defaults.
Synchronize parquet.rst Sphinx doctest expected row group byte size for V2 headers.
…ion in TestSerialize and synchronize Sphinx doctest sizes Update TestSerialize in Parquet C++ to allow selective compression leaving small or incompressible pages uncompressed under V2 page defaults. Wrap comment to adhere to 90-char limit for clang-format. Synchronize column chunk total_compressed_size and total_uncompressed_size in parquet.rst Sphinx doctest for V2 page defaults.
…V2 data pages Update TestSerialize in Parquet C++ to check positivity of compressed and uncompressed sizes under V2 data pages rather than asserting strict ordering. Under V2, selective compression can leave data pages uncompressed, while unconditional compression of dictionary pages can expand small payloads, making total_compressed_size less than, equal to, or greater than total_byte_size.
adamreeve
left a comment
There was a problem hiding this comment.
Thanks for this @Awesome06! I have a couple of questions
…atistics with nulls Add TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLength to verify unreferenced dictionary values are not included in statistics when null count causes unique count to equal dictionary length. Add TestArrowWriteDictionary.NestedDictionaryParentNullStatistics to verify that nested dictionary columns under nullable parents accurately reflect ancestor nulls in page and chunk statistics across both V1 and V2 data pages.
d61267d to
fd23951
Compare
… nested dictionary test
|
@Awesome06 Please rewrite your PR description as a human writing for humans. |
|
@pitrou Kindly let me know the changes you expect in the PR description. As I'm not quite sure what to reduce or reword in the current description |
|
@Awesome06 I think my demand was quite self-descriptive. If you are a human, write the PR description yourself instead of pasting AI output. If you are not a human, please say so and this PR will be closed. |
| @@ -2024,7 +2024,8 @@ Status TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary( | |||
| ::arrow::compute::Unique(*chunk_indices, &exec_ctx)); | |||
|
|
|||
| // On first run, we might be able to re-use the existing dictionary | |||
There was a problem hiding this comment.
Let's update this comment? This isn't about a first run but whether all dict values are referenced.
| // On first run, we might be able to re-use the existing dictionary | |
| // Use the original dictionary if we can detect that all indices are referenced, | |
| // otherwise call Take to find referenced dictionary values. |
| if (chunk_indices->null_count() == 0 && | ||
| referenced_indices.length() == dictionary->length()) { |
There was a problem hiding this comment.
Would it make sense to refine the shortcut condition? Something like:
| if (chunk_indices->null_count() == 0 && | |
| referenced_indices.length() == dictionary->length()) { | |
| const bool all_indices_referenced = | |
| (referenced_indices.null_count() == 0) | |
| ? referenced_indices.length() == dictionary->length() | |
| : referenced_indices.length() == dictionary->length() + 1; | |
| if (all_indices_referenced) { |
| } else { | ||
| // In Parquet V2, selective compression may keep small or incompressible data | ||
| // pages uncompressed (total_compressed_size == total_byte_size), while | ||
| // compressed dictionary pages or pages with savings may be larger or smaller |
There was a problem hiding this comment.
They shouldn't be "larger or smaller", they should be smaller, precisely because of selective compression.
| Using a larger value with V1 or a number outside the allowed range | ||
| raises ``ValueError``. |
There was a problem hiding this comment.
| Using a larger value with V1 or a number outside the allowed range | |
| raises ``ValueError``. | |
| Using a larger value with V1 data pages or a number outside | |
| the allowed range raises ``ValueError``. |
Rationale for this change
Parquet V2 data pages have been supported in Apache Arrow for several years, offering key improvements including per-page selective compression (
min_compression_savings) and page-level null count headers. This PR switches the default data page format from V1 to V2 across C++ and PyArrow, while retaining full support for writing V1 pages when requested.What changes are included in this PR?
data_page_versionfromV1toV2inWriterProperties(C++) and PyArrow (parquet.write_table,ParquetWriter,ParquetFileWriteOptions).TypedColumnWriterImpl::WriteArrowDictionarysodictionaryis reused when all dictionary entries are referenced even if nulls are present in the chunk.type_fwd.handparquet/core.pydescribing V2 as an opt-in feature.Are these changes tested?
Yes. Existing Parquet C++ and PyArrow test suites cover V2 page writing and reading. New unit tests were added:
TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLengthTestArrowWriteDictionary.NestedDictionaryParentNullStatisticstest_min_compression_savings_default_v2Are there any user-facing changes?
Yes. Parquet files written with default writer properties will now emit V2 data pages rather than V1 data pages. Applications requiring V1 data pages for legacy readers can set
data_page_version="1.0".Was AI used for this PR?
PR code and description written by:
Reviewed before submission by: